High Performance Training of Feedforward & Simple Recurrent Networks

TRAINREC is a system for training feedforward and recurrent neural networks that incorporates several ideas. It uses the conjugate-gradient method which is demonstrably more efficient than traditional backward error propagation. We assume epoch-based training and derive a new error function having several desirable properties absent from the traditional sum-of-squares-error function. We argue for skip (shortcut) connections where appropriate and the preference for a bipolar sigmoidal yielding values over the [-1,1] interval. The input feature space is often over-analyzed, but by using singular value decomposition , input patterns can be conditioned for better learning often with a reduced number of input units. Recurrent networks, in their most general form, require special handling and cannot be simply a rewiring of the architecture without a corresponding revision of the derivative calculations. There is a careful balance required among the network architecture (specifically, hidden and feedback units), the amount of training applied, and the ability of the network to generalize. These issues often hinge on selecting the proper stopping criterion. Discovering methods that work in theory as well as in practice is difficult and we have spent a substantial amount of effort evaluating and testing these ideas on real problems to determine their value. This paper encapsulates a number of such ideas ranging from those motivated by a desire for efficiency of training to those motivated by correctness and accuracy of the result. While this paper is intended to be self-contained, several references are provided to other work upon which many of our claims are based. The popularity of neural networks continues to increase, particularly as researchers have realized the importance of recurrent networks. Recurrent architectures have the advantage of being able to remember, to some degree, what inputs or events have occurred earlier in a sequence and are able to exert control of future decision-making based on these past events. For example Servan-Schreiber et. al.[26] discuss usage of recurrent networks as representations of state machines. The challenges of recurrent networks are similar but not precisely the same as those for ordinary, feedforward networks. Neural networks, as a field, is graduating from the novelty and toy problem stage and is becoming an important and useful tool for a wide variety of problem areas. As our understanding of the methodology increases through research, we become better equipped to address tasks on the scale of practical, real-world problems. Likewise, it seems, as our ability to train …